Method and system for assessment of cognitive function based on electronic device usage

ABSTRACT

A system and method that enables a person to unobtrusively quantify the effect of mobility, physical activity, learning, social interaction and diet on cognitive function. The method records on the electronic device one of global positioning system longitude and latitude coordinates, accelerometer coordinates, and gyroscope coordinates, one of outgoing and incoming phone calls, outgoing and incoming emails, and outgoing and incoming text messages, one of URLs visited on an internet browser application, books read on an e-reader application, games played on game applications, and the nutritional content of food consumed, performs the step of learning a function mapping from those recordings to measurements of cognitive function using a loss function to identify a set of optimal weights that produce a minimum for the loss function, uses those optimal weights to create the function mapping, and performs the step of computing the variance of the cognitive function measurements that is explained by the function mapping to assign an attribution to the effect of physical activity on measured changes in cognitive function.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of pending U.S. patentapplication Ser. No. 14/059,682 filed Oct. 22, 2013 entitled METHOD ANDSYSTEM FOR ASSESSMENT OF COGNITIVE FUNCTION BASED ON MOBILE DEVICEUSAGE, which is hereby incorporated by reference in its entirety.

BACKGROUND

1. Field of Art

The invention relates generally to computing a person's cognitivefunction and, more specifically, to unobtrusive assessment of cognitivefunction from electronic device usage.

2. Background Art

Cognitive function tests measure a person's cognitive abilities across abroad range of cognitive domains such as memory (working memory,semantic memory, episodic memory), attention, processing speed(visuospatial, symbol substitution), verbal skills, generalintelligence, and executive function. Today, cognitive function testsare administered by a trained psychometrician requiring several hours oftesting and cannot be repeated more frequently than once per year. Thetest scores can vary due to a change in the person's cognitive function,due to the subjective nature of the interpretation of the tests, or dueto situational factors that may affect an individual on the day of thetest.

Brain health is critical to our success as individuals in anincreasingly cognitive demanding society. In school aged children andadolescents, brain health is responsible for academic success. Inworking individuals, brain health leads to improved job performance, andin the elderly it enables autonomy, independence and greater enjoymentfrom activities.

Cognitive function is a measure of brain health, and factors that affectthe brain also affect cognitive function. These factors can becategorized into situational, traumatic, and disease related.Situational factors include lifestyle decisions on diet, socialengagement, intellectual stimulation, physical activity, sleep patterns,and stress levels. For example, during periods of high stress, poorsleep, and inadequate physical activity, a person will perform worse ona cognitively demanding task [1]. Unfortunately, there is no knownreliable system or method for repeated and regular assessment ofcognitive function to inform a person of the harm or benefit thatcurrent lifestyle decisions have on their brain health. Repeat cognitivefunction testing by a psychometrician is neither practical nor reliablewhen repeated more frequently than once per year because the individualacquires test-taking skills for the test. Similarly, the emergence ofonline tests available through many application vendors such asBrainBaseline [2] suffer test practice effects that are well documented[3,4] whereby the subject develops test taking skills that increasetheir scores but do not transfer well to real world activities andundermine the test's sensitivity and specificity to cognitive changes.

Traumatic factors affecting cognitive function include blunt orpenetrating head injuries. Unlike situational factors, the effect oftraumatic brain injury on cognitive function is not generallyreversible. Traumatic brain injury is increasingly recognized as acontributor to cognitive function deficits in players of contact sports.Early detection of changes in cognitive function for contact sportathletes is paramount to their brain health and requires a systemicmethod to measure cognitive function that is repeatable, reliable, andunobtrusive.

Many diseases are known to affect brain health and cognitive function.The progression of aging into mild cognitive impairment and Alzheimer'sdisease is of great societal concern because of its rapidly increasingprevalence in an increasingly older society. Today, one in eight olderAmericans has Alzheimer's disease and Alzheimer's is the sixth leadingcause of death in the United States [5]. By 2025, the number ofAmericans age 65 and older with Alzheimer's disease is estimated toincrease 30%, and by 2050 that number is expected to triple, barring anybreakthroughs to prevent, slow or arrest the disease [5]. Prior todeveloping Alzheimer's disease, patients go through a six-year prodromalphase of cognitive decline. The societal burden of mental disease in theelderly is staggering and poised to worsen. A repeatable, reliable, andunobtrusive test of cognitive function is needed to monitor brain healthin aging adults to enable early detection and intervention.

Mood disorders include depressive disorders, bipolar disorders, andsubstance-induced mood disorders. They affect people of all ages andimpair multiple cognitive domains [6]. In the United States, mooddisorders are among the most common reason for hospitalization inchildren under 18 [7]. With a 12-month adult prevalence of 9.5% [8],mood disorders are a costly social burden. Mood disorder therapy focuseson the behavioral and mood related facets of the disease to thedetriment of the cognitive function deficits. Medications to treat mooddisorders can worsen cognitive domains such as memory. The impact ofcognitive deficits in a person's school or job performance can besignificant, is exacerbated by treatment, and frequently unrecognizeddue to lack of adequate repeatable, reliable, and unobtrusive measuresof cognitive function.

Many other diseases are known to affect brain health and cognitivefunction. These include neurovascular disorders including multi-infractdementia, hepatic failure with encephalopathy, renal failure, congestiveheart failure, and various infectious disease and viral illness to namea few. Individuals with any of these disorders are at risk for cognitiveimpairment and would benefit from repeatable, reliable, and unobtrusivemeasures of cognitive function.

The introduction of mobile devices and their broad adoption hasrevolutionized how society interacts both with each other and with theirsurroundings. A smartphone today enables a user to make calls, send andreceive emails and text messages, find their location on a map orretrieve directions to a destination point, browse the internet,download and play game applications, and a host of other activities. Inaddition, these smartphones are equipped with accelerometers andgyroscopes that sense the device's acceleration and orientation in3-dimensions. Processing of the acceleration and orientation signalsreveals the user's activity such as whether the person is walking orjogging. Mobile devices encompassing wearable electronic devices such aswatches, clothing, and glasses [9,10] are also capable of deliveringmuch of the functionality found in a smartphone.

One company that has leveraged the close interaction of an individualwith their mobile device to make behavioral assessments is Ginger.io[11,12]. Ginger.io provides a smartphone application that tracks thenumber and frequency of calls, text messages, and emails sent, and usesthe device's global positioning system (GPS) and accelerometer to inferactivity level. The target population for Ginger.io's application ispatients with chronic diseases such as diabetes, mental disorders, andCrohn's disease. When a patient deviates from their routine behavior ofcalling and texting patterns, Ginger.io alerts the individual'scaregiver to intervene and assess the situation for noncompliance withmedications, inappropriate titration of medications, and other factorsthat may precipitate a flare-up of the patient's disease. This approachis behaviorally based and does not measure cognitive function but ratherchanges in behavior that may be attributed to a preexisting disorderflare up.

Recent research supports a close interaction between motion andcognition, largely mediated by interconnections between the cerebellumresponsible for motion and areas in the brain such as the prefrontalcortex responsible cognition [13,14]. Sensors in an electronic device,including wearable devices, provide insights into the user's motion andenable detection of irregularities or changes in motion.

What is needed is a method and system to assess cognitive function thatis repeatable, reliable, and unobtrusive to an individual.

-   -   1. Lieberman H R, Thario W J, Shukitt-Hale B, Speckman K L,        Tulley R, Effects of caffeine, sleep loss, and stress on        cognitive performance and mood during U.S. Navy SEAL training.        Psychopharmacology, 2002, 164:250-261    -   2. www.brainbaseline.com    -   3. Ackerman P L, Individual differences in skill learning: An        integration of psychometric and information processing        perspectives. Psychol Bull, 1987, 102:3-27    -   4. Healy A F, Wohldmann E L, Sutton E M, Bourne L E, Jr,        Specificity effects in training and transfer of speeded        responses. J Exp Psychol Learn Mem Cognit, 2006, 32:534-546    -   5. Alzheimer's Association, 2012 Alzheimer's Disease Facts and        Figures. www.alz.org/downloads/facts_figures_2012.pdf    -   6. Marvel, Cherie L., and Sergio Paradiso. “Cognitive and        neurological impairment in mood disorders.” The Psychiatric        clinics of North America 27.1 (2004): 19.    -   7. Pfuntner A., Wier L. M., Stocks C. Most Frequent Conditions        in U.S. Hospitals, 2011. HCUP Statistical Brief #162.        September 2013. Agency for Healthcare Research and Quality,        Rockville, Md.    -   8. Kessler R C, Chiu W T, Demler O, Walters EE. Prevalence,        severity, and comorbidity of twelve-month DSM-IV disorders in        the National Comorbidity Survey Replication (NCS-R). Archives of        General Psychiatry, 2005 June; 62(6):617-27.    -   9. http://www.crunchwear.com    -   10. http://en.wikipedia.org/wiki/Google_Glass    -   11. www.ginger.io    -   12. Owen Covington, ‘Virtual nurse’ helps Forsyth Medical Center        track diabetics. The Business Journal, May 2013,        http://www.biziournals.com/triad/news/2013/05/20/forsyth-medical-center-using-virtual.html    -   13. Koziol L F, Budding D, Andreasen N, D'Arrigo S, Bulgheroni        S, et al. (2013) Consensus Paper: The Cerebellum's Role in        Movement and Cognition. Cerebellum: Cerebellum (2014) 13:151-177    -   14. Jensen E. Teaching with the Brain in Mind, 2^(nd) Edition.        Association for Supervision & Curriculum Deve; Revised 2nd        edition, 2005

BRIEF SUMMARY OF INVENTION

The invention enables a person to monitor changes in their or another'scognitive function in an unobtrusive manner, to view those changes overtime, and to evaluate the impact on that changes in social engagement,physical activity, learning activity, and diet have on their cognitivefunction evaluation. What is needed is a method and system to assesscognitive function that is highly sensitive, specific, and unobtrusiveto an individual. Such a method and system would measure and track aperson's cognitive function without explicit input or behavioral changesrequired by the subject, such as repeated neuropsychological evaluationsand online tests. Rather, the method and system would use digitallyrecorded interactions of an individual with their electronic devices,such as mobile devices and wearable electronics, to compute measures ofcognitive function, detect changes in cognitive function, and inferattribution to changes in behavioral activity without disrupting theuser's day-to-day activities or their use of electronic devices.

One embodiment of the present invention is a method for unobtrusivelyrecording an individual's interaction with an electronic device that mayinclude applications opened, inputs typed, gesture patterns used on atouch-screen, body motions, eye-movements and voice input. The method ofthe present invention may include the step of recording data from theelectronic device's global positioning system (GPS), accelerometer, andgyroscope to infer daily activity including the activity intensity,daily mobility including method of travel, and daily social engagementthrough latitude and longitude localization of travel destination to ashopping center, a museum, or a restaurant. This data may provideinsight regarding an individual lifestyle, including their socialskills, level of activity and dietary habits. These insights maycontribute to good health and/or they may be indicative of a problem.The method of the present invention may further include the step ofrecording data from the electronic device's phone, email, and textingapplications to capture incoming and outgoing calls, emails and textsgenerated, length of conversation, length of messages, and discrepanciesin voice messages and email messages opened versus received, which areused as additional inputs to infer changes in social engagement.

The method of the present invention can further include the step ofrecording data from a barcode scanning application used to scanpurchased grocery items, food and beverages consumed, and supplementingthat data with nutritional fact information to track diet attributessuch as caloric input, calories from fat, consumed saturated fats, transfats, cholesterol, sugars, protein, vitamins and minerals. The method ofthe present invention may also include the step of recording medicationstaken, dosages, and quantities. This information is also indicative oflifestyle that is healthy or not. This information can also correlate toan increase or decline in an individual's cognitive function. The methodof the present invention can further include the step of recording datafrom wearable devices that measure, by way of example, heart-rate, bloodoxymetry, body temperature, electroencephalogram, and communicate thatinformation to an application resident on a mobile device to infer theuser's physical activity, activity intensity, and learning activity.This information related to an individual's biological vitals mayexplain why there is a change in cognitive function and why the changeis not problematic and/or this information can be an indicator of asystemic problem that will have a long term negative impact on cognitivefunction. The method may further include the step of recording the URLsvisited on an internet browser application, e-book pages read on ane-book application resident on the electronic device, the contentclassification of the material read and its level of complexity, and thelanguage of the content, to further infer learning activity by the user.

The data captured from the user's electronic device by the method of thepresent invention is preferably persisted in the device's storage andfurther transmitted to a cloud computing system to compute cognitivefunction from the user's interactions and to infer behavioral activityattribution effects on changes in cognitive function. The cognitivefunction assessment and behavioral activity attributions are preferablypresented to the user in a password protected online portal that revealspositive and negative contributors to trends in cognitive functionassessment and establishes behavioral targets to improve cognitivefunction. Those targets, and any ensuing improvement or decline incognitive function may be subsequently measured by the method of thepresent invention, enabling an unobtrusive, closed-loop method andsystem for assessing and improving cognitive function from electronicdevice usage. The cloud computing environment preferably allows a userto change electronic devices and/or electronic device service carriersand still have access to previously recorded data.

These and other advantageous features of the present invention will bein part apparent and in part pointed out herein below.

BRIEF DESCRIPTION OF THE DRAWING

For a better understanding of the present invention, reference may bemade to the accompanying drawings in which:

FIG. 1 illustrates an embodiment of the unobtrusive cognitive functionassessment system configured in accordance with the present invention;

FIG. 2 is a functional description of the unobtrusive cognitive functionassessment system in accordance with one embodiment of the presentinvention;

FIG. 3 illustrates an exemplary computing environment of the unobtrusivecognitive function assessment system configured in accordance with oneembodiment of the present invention;

FIG. 4 is a data collection module in accordance with one embodiment ofthe present invention;

FIG. 5 is a transmission module in accordance with one embodiment of thepresent invention;

FIG. 6(a)-(b) is a feature extraction module in accordance with oneembodiment of the present invention;

FIG. 7(a)-(c) is a metric computation module in accordance with oneembodiment of the present invention; and

FIG. 8 is a reporting module in accordance with one embodiment of thepresent invention.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and will herein be described in detail. Itshould be understood, however, that the drawings and detaileddescription presented herein are not intended to limit the invention tothe particular embodiment disclosed, but on the contrary, the intentionis to cover all modifications, equivalents, and alternatives fallingwithin the spirit and scope of the present invention as defined by theappended claims.

DETAILED DESCRIPTION

According to the embodiment(s) of the present invention, various viewsare illustrated in FIG. 1-8 and like reference numerals are being usedconsistently throughout to refer to like and corresponding parts of theinvention for all of the various views and figures of the drawings.

The following detailed description of the invention contains manyspecifics for the purpose of illustration. Any one of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within scope of the invention. Accordingly, thefollowing implementations of the invention are set forth without anyloss of generality to, and without imposing limitations upon, theclaimed invention.

One implementation of the present invention comprises a system andmethod that enables an unobtrusive assessment of a person's cognitivefunction from electronic device usage, and in which an embodimentpreferably teaches a novel system and method for recording on theelectronic device the occurrence and timing of user events comprisingthe opening and closing of applications resident on the device, thecharacters inputted and the touch-screen gestures, tapping, bodymotions, and eye movements used on those applications, further recordingthe kinetic activities of motion, gait and balance from wearablegyroscopic and accelerometer sensors. The embodiment further preferablyincludes performing the step of learning a function mapping from theelectronic device recordings to measurements of cognitive function for apopulation of users that uses a loss function to determine relevantfeatures in the recording, identifies a set of optimal weights thatproduce a minimum of the loss function, creates a function mapping usingthe optimal weights. Further, the embodiment may include performing thestep of applying the learned function mapping to a new recording fromthe electronic device of new users or from the same population of usersto compute a new cognitive function value. For the cognitive function, alinear or nonlinear functional model can be used that inputs therecorded activity by the electronic device, and subsequently outputs acognitive function measurement. In one embodiment, the cognitivefunction model is estimated, or fit, to the data recorded fromelectronic devices for a population of users for which baselinecognitive function measurements are available. The cognitive functionmodel is then applied to new recordings from the electronic devices ofnew users or from the same population of users to compute a newcognitive function value. In another embodiment, the cognitive functionof a user is the percentile rank of a metric computed from the datarecorded from the user's electronic device when compared to all users.Diet, medications, stress, chronic lack of sleep, disease, and mooddisorders such as depression can all temporally affect cognitivefunction and would be reflected in repeated evaluations of the cognitivefunction model upon receiving input data from a user's electronicdevice. Powerful nonlinear models include neural networks and deepbelief networks, a machine learning function composed of layers ofneural networks capable of learning complex high-level features. Asoftware application can reside on an electronic computing device, suchas a smart phone, personal data assistant (PDA), tablet computer, orwearable electronics such as glasses, watches, or clothing such thatwhen it is executed by the processor of the computing device, the stepsof the method may be performed.

Another implementation of the present invention comprises a system andmethod that enables a person to unobtrusively quantify the effect ofphysical activity on cognitive function, in which the embodiment teachesa novel system and method for repeatedly recording on the electronicdevice one of global positioning system longitude and latitudecoordinates, accelerometer coordinates, and gyroscope coordinates. Theimplementation further performs the step of learning a function mappingfrom those recordings to measurements of cognitive function using a lossfunction to identify a set of optimal weights that produce a minimum forthe loss function, uses those optimal weights to create the functionmapping, and the embodiment further includes performing the step ofcomputing the variance of the cognitive function measurements that isexplained by the function mapping to assign an attribution to the effectof physical activity on measured changes in cognitive function. Forexample, a mixed model function is a type of function containing bothfixed and random effects and is used in settings where repeatedmeasurements are made over time on the same subjects. The mixed modelfunction can be utilized for the learned function mapping, whichdevelops the appropriate attribution physical activity levels to thecognitive function from the repeated measurements.

A further implementation of the present invention comprises a system andmethod that enables a person to unobtrusively quantify the effect ofsocial activity on cognitive function, which embodiment teaches a novelsystem and method for repeatedly recording on the electronic device oneof outgoing and incoming phone calls, outgoing and incoming emails, andoutgoing and incoming text messages. The implementation further includesperforming the step of learning a function mapping from those recordingsto measurements of cognitive function using a loss function to identifya set of optimal weights that produce a minimum for the loss function,uses those optimal weights to create the function mapping, and theembodiment further includes performing the step of computing thevariance of the cognitive function measurements that is explained by thefunction mapping to assign an attribution to the effect of socialactivity on measured changes in cognitive function.

A further implementation of the present invention comprises a system andmethod that enables a person to unobtrusively quantify the effect oflearning activity on cognitive function, which embodiment teaches anovel system and method for repeatedly recording on the electronicdevice one of URLs visited on an internet browser application, booksread on an e-reader application, and/or games played on gameapplications. The implementation further may include performing the stepof learning a function mapping from those recordings to measurements ofcognitive function using a loss function to identify a set of optimalweights that produce a minimum for the loss function and uses thoseoptimal weights to create the function mapping. The embodiment furtherincludes performing the step of computing the variance of the cognitivefunction measurements that is explained by the function mapping toassign an attribution to the effect of learning activity on measuredchanges in cognitive function.

A further implementation of the present invention comprises a system andmethod that enables a person to unobtrusively quantify the effect ofdiet and medications on cognitive function, which the embodiment teachesa novel system and method for repeatedly recording on the electronicdevice the nutritional content of food consumed, caloric intake by foodgroup, alcohol, caffeine and medications taken, dosages, and frequencythereof. The implementation further includes performing the step oflearning a function mapping from those recordings to measurements ofcognitive function using a loss function to identify a set of optimalweights that produce a minimum for the loss function and uses thoseoptimal weights to create the function mapping. The embodiment furtherincludes performing the step of computing the variance of the cognitivefunction measurements that is explained by the function mapping toassign an attribution to the effect of diet and medications on measuredchanges in cognitive function.

The loss function for the cognitive function can evaluate baselinerecordings and changes and trends in the recordings both from aquantitative and qualitative perspective. For example the frequency ofevents can be monitored for cognitive measurement as well as the qualityof each event, such as latencies between gestures, tapping, bodymotions, eye movements or keystrokes, erroneous keystrokes or gestures,duration of key presses, misspellings and kinetic activities of motion,gait, or balance that are captured by wearable electronic sensors andrecorded or transmitted to an electronic device.

Another implementation of the invention can be a computer systemcomprising a mobile computer (for example a smart-phone; tablet or PDAcomputing device; wearable electronic accessories such as glasses orwatches and wearable electronic clothing) including a wireless networkinterface which communicates over a wireless network with a secondcomputer including a network interface, each computer further includinga processor, a memory unit operable for storing a computer program, aninput mechanism operable for inputting data into said computer system,an output mechanism for presenting information to a user, a bus couplingthe processor to the memory unit, input mechanism and output mechanism,and wherein the mobile computer system includes various executableprogram modules stored thereon where when executed are operable toperform functions.

The computer system may comprise a data collection module stored on amobile computer where, when executed, records to the memory unit of saidmobile computer the occurrence and timing of user events comprising theopening and closing of applications resident on said mobile computer,the characters inputted in said applications, the touch-screen gestures,tapping, body motions, and eye movements made on said applications. Thetype of the event and both the frequency and timing of the events andthe various qualitative metrics regarding the event are recorded. Atransmission module can also be stored on said mobile computer that whenexecuted transmits through the wireless network interface the recordingsstored in the memory unit to said second computer. A feature extractionmodule and metric computation module can be stored on said secondcomputer that when executed learns a function mapping from saidtransmitted recording to measurements of cognitive function using a lossfunction to determine relevant features (including frequency, timing andqualitative metrics) in said recording, identifies a set of optimalweights that produce a minimum for said loss function, and creates saidfunction mapping using said optimal weights. After learning the functionmapping, the same feature extraction and metric computation modules whenexecuted applies said function mapping to a new transmitted recording ofthe occurrence and timing of events comprising the opening and closingof applications resident on said device, the characters inputted in saidapplications, the touch-screen gestures, tapping, body motions, and eyemovements used on those applications, the kinetic activities of motion,gait and balance recorded from wearable gyroscopic and accelerometersensors, to calculate a new cognitive function value.

Another implementation of the invention can be a computer systemcomprising a mobile computer including a wireless network interfacewhich communicates with a second computer including a network interface,each computer further including a processor, a memory unit operable forstoring a computer program, an input mechanism operable for inputtingdata into said computer system, an output mechanism for presentinginformation to a user, a bus coupling the processor to the memory unit,input mechanism and output mechanism, wherein the mobile computer systemincludes various executable program modules stored thereon where whenexecuted are operable to perform functions.

The computer system can comprise a motion module stored on said mobilecomputer where when executed records to the memory unit of said mobilecomputer one of global positioning system longitude and latitudecoordinates, accelerometer coordinates, and gyroscope coordinates. Inanother implementation, the computer system can comprise a social modulestored on said mobile computer that when executed records to the memoryunit of said mobile computer one of outgoing and incoming phone calls,outgoing and incoming emails, and outgoing and incoming text messages.In another implementation, the computer system may comprise a learningmodule stored on said mobile computer that when executed records to thememory unit of said mobile computer at least one of: URLs visited on aninternet browser application, books read on an e-reader application, andgames played on game applications. In another implementation, thecomputer system may comprise a diet and medication module stored on saidmobile computer that when executed records nutritional content of foodconsumed, caloric intake by food group, medications taken, dosages andfrequencies thereof.

In any of the preceding implementations, a transmission module can bestored on said mobile computer that when executed transmits through thewireless network interface the recordings stored in the memory unit tosaid second computer. In any of the preceding implementations, anattribution module can be stored on said second computer that whenexecuted learns a function mapping from said transmitted recording tomeasurements of cognitive function using a loss function to identify aset of optimal weights that produce a minimum for said loss function,and creates said function mapping using said optimal weights. In any ofthe preceding implementations, a reporting module can be stored on saidsecond computer where when executed presents to the user the computedcognitive function metrics and the attribution to said metrics that areexplained by said recordings.

The details of the invention and various embodiments can be betterunderstood by referring to the figures of the drawing. FIG. 1illustrates an embodiment of the functional description of the systemconfigured in accordance with the present invention and is not intendedto limit scope as one of ordinary skill would understand on review ofthis application that other configurations could be utilized withoutdeparting from the scope of the claimed invention. Referring to FIG. 1an individual uses his or her electronic device 101 which may be a smartphone or tablet computer; a wearable electronic device such aselectronic glasses, electronic wristband device, or smart clothing; or ahousehold electronic device such as a desktop computer or a remotecontrol. In 103, the individual taps a touch-screen, speaks, uses atouch pad, or invokes an external device such as mouse to open anapplication; uses touch-screen gestures, eye movements, body motionsincluding head tilting and swiping, or invokes an external device suchas mouse to scroll within an application; types messages, directions,and other content on a keyboard exposed by the application or uses voicecommands to do the same; reads and scrolls through content, and makescalls or listens to messages. The user's activity in 103 is recorded onthe device's persistent storage 113. The storage 113 being of the typegenerally some known in the art. In 105, the user makes or receivesphone calls, email messages, and text messages in a manner known in theart.

The date and time of a user's activity, the activity duration, and thesender and recipient phone number or email address are preferablyrecorded in the device's persistent storage 113. The frequency of eachevent may be recorded and the various qualitative characteristics ofeach event may also be recorded. In 107, when the electronic device isalso a mobile or wearable device, the user preferably carries theelectronic device with them while driving, using public transportation,walking, biking or running or when the user is otherwise in motion.While engaging in these activities, the device's global positioningsystem (GPS) records the user's longitude and latitude coordinates in107. Similarly, in 107, the device's acceleration and gyroscopic motionalong a 3-coordinate system is preferably recorded. The type oflocations to which the individual traveled may be determined and thecharacteristic of the motion of the user may also be evaluated forfluidity or erratic motion. This information is also preferably recordedin the device's persistent storage 113. In 109, the user may browse URLson an internet-browser application resident on the electronic deviceand/or reads an e-book on an e-book reader resident on the device. TheURLs browsed and/or the pages of the e-book read, the start time and endtime between URLs and pages may be recorded by 109 and persisted in 113.

Note that all gesture activity, typing activity, and voice commandactivity during the use of applications tracked in 105, 107, and 109 ispreferably captured separately in 103 and recorded with the time andapplication in which that activity took place. In this way, the systemtracks gestures used during browsing, paging and scrolling, for example.Lastly, in 111 a bar code scanning application and medical reminderapplication that may be resident on the electronic device preferablyenables the user to scan grocery purchases and/or meals and beveragespurchased when the latter have bar codes. The bar code scanningapplication preferably has access to a database of nutritional facts. Ifthe nutritional facts for the scanned bar code is not in the database,then in 111 the application may instruct the user to photograph thenutritional fact label of the item. The bar code information and themedications taken, dosages, and frequencies are also preferablypersisted in 113.

The data captured in the device's persistent storage 113 may betransmitted to cloud computers 115. Transmission preferably uses asecure channel and can use hypertext transfer protocol secure (HTTPS) tosecurely transfer the data to 115. The cloud computers 115 may analyzethe recorded data against historical recordings and against recordingsof other users including users that are demographically matched to theexisting user. The outputs of the analyses are preferably one or morecognitive function measures. Further outputs may include the effect thatbehavioral activities, inferred from the activities recorded in 105-111,had on the various cognitive function measures. These behavioralactivities include social engagement 105, physical activity 107,learning activity 109, and/or diet and medications 111. The user or theuser's delegate in 117 may log into a password protected online accountto view these outputs.

In another embodiment of the invention the outputs of the analyses bythe cloud computers 115 are transmitted back to the device's persistentstorage 113. The user views the outputs directly in the display screenof the electronic device 101.

In yet another embodiment of the invention, the metric computationmodules of FIG. 7(b) or FIG. 7(c) and described herein below areresident on the electronic device 113. In this embodiment, the cognitivefunction metrics are computed locally on the device and the user viewsthe outputs directly in the display screen of the electronic device 101.The outputs may also be transmitted to the cloud computer 115, and theuser or the user's delegate in 117 may log into a password protectedonline account to view these outputs.

In one implementation of the method and system, in order to establish abaseline of data, supervised benchmark testing can be conducted on aninitial test group of individuals where these individuals take aneuropsychological benchmark test for cognitive function, and the datais stored. Each of the same individuals who are tested can be providedwith mobile devices having the computer program for performing thesystem and method described herein. Data for each individual can berecorded as outlined herein, and the data from the mobile device usagecan be correlated to the benchmark testing results and cognitivefunction. Cognitive function levels and bands may also be determinedfrom the result. Once certain baselines have been established andcorrelations are made between cognitive function and mobile deviceusage, all subsequent mobile device usage by individuals may be utilizedto improve the system and method as learning occurs. The learning fromthe subsequent mobile device usage may be considered unsupervisedlearning.

FIG. 2 illustrates an embodiment of the functional description of thesystem configured in accordance with the present invention and is notintended to limit scope, as one of ordinary skill would understand onreview of this application that other configurations could be utilizedwithout departing from the scope of the claimed invention. Referring toFIG. 2, a user's interaction with an electronic device may be capturedand recorded by the data collection module 400 resident on the device.The transmission module 500 is preferably resident on the device andresponsible for transmitting and receiving data to and from the cloudcomputers. This module preferably uses broadband WIFI for thetransmission when available, but may alternatively use othertransmission means including 4G LTE transmission provided by subscriberdata plans. Transmission module 500 preferably is also responsible forsecuring an encrypted channel.

The feature extraction module 600 may extract patterns and features fromthe data acquired by one or more of the user's electronic devices usingthe data collection module 400 resident on each device. In oneembodiment of the present invention, feature module extraction 600 isresident on cloud computers. In another embodiment of the presentinvention, feature extraction module 600 is resident on the deviceitself, and features are extracted from data acquired by data collectionmodule 400 on that device. The metric computation module 700 maymaintain predictive models of cognitive function measures that it takesas input features from feature extraction module 600. In one embodimentof the present invention, the predictive models of metric modulecomputation 700 are resident on cloud computers. In another embodimentof the present invention, the predictive models of metric modulecomputation 700 and the feature extraction module 600 are both residenton the device itself and take as input features extracted from datacollected on that device. The metric computation model 700 may also beresponsible for learning the predictive models of cognitive functionmeasures from a population of users for which both feature data andtraditional cognitive function testing data is available.

Metric module computation 700 may also compute the attribution to theuser's cognitive function measures of the user's behavioral activitiesinferred from data acquired by data collection module 400. A reportingmodule 800 preferably provides an online login account for the user orthe user's delegate to review trends in the user's cognitive functionmeasures, attribution of recent behavior activity to those trends, howwell the user is tracking to target behavior activity and targetcognitive function, and further may enable the user to update thosetargets.

FIG. 3 illustrates an embodiment of the computing environment of thesystem configured in accordance with the present invention and is notintended to limit scope as one of ordinary skill would understand onreview of this application that other configurations could be utilizedwithout departing from the scope of the claimed invention. Referring toFIG. 3, a user may download and install a software application to theirelectronic device 301 from a computer associated with the cloud 305. Inone embodiment of the present invention, the software applicationcomprises the data collection module 400 and the transmission module500. These two modules 400, 500 preferably read and write to theelectronic device's persistent storage 303. The data collection module400 may write raw activity data to the persistent storage 303. Thetransmission module 500 may transmit the data persisted in storage 303to the cloud computers 305. The feature extraction module 600, metriccomputation module 700, and reporting module 800 are preferably residentin the cloud computers 305. Feature extraction module 600 may extractpatterns and features from the data that are used as inputs to themetric computation module 700. The metric computation module 700 mayoutput the cognitive function measures. Metric computation module 700may also output the attribution of the user's behavioral features to thecognitive function measures. The reporting module 800 may create summaryreports presented to the user or the user's delegate in apassword-protected account on the cloud servers.

In another embodiment of the present invention, the software applicationcomprises the data collection module 400, the transmission module 500,the feature extraction module 600, and the metric computation module700, and the reporting module 800. Modules 400, 500, 600, 700, 800 arepreferably resident on the electronic device. The data collection module400 may write raw activity data to the persistent storage 303. Featureextraction module 600 may extract patterns and features from the datalocally on the device, and metric computation module 700 may use thosefeatures to compute cognitive function metrics. Metric computationmodule 700 may also output the attribution of the user's behavioralfeatures to computed cognitive function metrics. The reporting module800 preferably displays to the user the outputs of metric computationmodule 700. Transmission module 500 may communicate with cloud servers305 and can optionally transmit the outputs of metric computation module700 to a cloud-based reporting module accessible to the user and theuser's delegates. Transmission module 500 may also download updates tothe software application.

Referring to FIG. 4, an electronic device such as smartphone, a tabletcomputer, desktop computer, wearable electronic clothing or accessoriesand other similar devices may have many applications available to theuser. These applications include, but are not limited to, maps used fordirections and global positioning localization, weather, reminders,clock, newsstand configured with user selected publications, mailconfigured by the user to include personal and work email, phoneconfigured by the user to include contacts, messages for texting, and aninternet browser.

Module 401 preferably tracks the application opened, the date-time thatit was opened, date-time that it was closed and records that informationon the device's persistent storage. Upon opening an application, a usermay interact with the application through one or more keyboard inputs,mouse inputs, voice, gestures, body motions, and eye movements. In 403,applications running on devices that support touch-screen gestures suchas tapping and swiping may be recorded. For applications running onwearable electronic devices, 403 records body motion such as headtilting, swiping, or eye movements. Module 403 may record the date-timeof the gesture, body motion or eye movement, the duration and latency,the application in use, and the application view change that resultsfrom the gesture, body motion, or eye movement. In 405, keyboard entriesmay be recorded for applications supporting a keyboard entry mode. In405, all keyboard entries are preferably recorded, which includes forexample alphanumeric entries, backspace entries, capitalization,formatting entries, and editing entries. Module 405 further may recordthe latency and duration of each keyboard entry, the application in use,and the application view change resulting from the key entry. Allrecordings are preferably stored in the device's persistent storage.

Module 407 may record words and phrases during phone conversations andvoice-input commands. Module 407 further may record the application inuse, and the application view change resulting from a voice command. Allrecordings are preferably stored in the device's persistent storage.Module 409 may record sensor data including gyroscope and accelerometerreadings from the electronic device, including wearable devices. Thisdata preferably provides information on user activity and kineticinformation including motion, gait, and posture when recorded fromwearable sensors. Recordings from a wearable electroencephalogram deviceare also preferably capture by 409. All recordings are preferably storedin the device's persistent storage.

Module 411 may record incoming and outgoing email, text messages, andcalls and further may record recipient and sender email addresses,recipient and sender text message phone numbers, and outgoing andincoming phone numbers. This module may further record words, phrases,and sentiment of the communication exchanges. The information ispreferably stored in the device's persistent storage together with thedate-time of the event.

For electronic devices equipped with a global positioning system (GPS),module 413 may sample the device's location, velocity, altitude, timestamp the input and store it in the device's persistent storage.

Recordings from wearable electronic devices that measure heart rate,blood pressure, pulse oxymetry, body temperature, and otherphysiological data may be time stamped and recorded in the device'spersistent storage in 415. The peripheral accessories can be used toobtain biological vital signs of the individual, which can be used todetermine if a decline in cognitive function data is due to a vital signsuch as fatigue or low blood-glucose levels rather than an actualdecline in cognitive function of the individual. The biological vitalscan also be used as an alert of a biological trend that will have along-term negative impact on cognitive function, such as hypertension.

Module 417 preferably enables a barcode scanning application enhancedwith nutritional fact information appended to each barcode entry, thatrecords and timestamps caloric input, nutritional content, alcohol, andcaffeine consumption. Medications taken, quantities and dosages maysimilarly recorded in the device's persistent storage. The dietaryinformation may be used as an alert of a dietary trend that will have along-term negative impact on cognitive function, such as high alcoholintake or high fat or cholesterol intake. The medication information maybe used to determine the negative or beneficial effect that medicationsand dosages have on cognitive function.

Referring to FIG. 5, the transmission module may be a background processthat runs on the electronic device and sleeps until it is awaken by therecording of new events on the device in 501. Upon recording newactivity, 501 may pass control to 503 that attempts to establish WIFIaccess. If 503 succeeds, then it may pass control to 507 to initiatetransmission of all recorded activity since the last successfultransmission. If 503 fails then it may pass control to 505 of thetransmission module which evaluates whether a successful transmissionoccurred within a 24 hour period. If 505 determines that no successfultransmission has occurred within a 24 hour period, it may pass controlto 507 to initiate transmission of all recorded activity since the lastsuccessful transmission. If, however, 505 confirms that a successfultransmission has occurred within 24 hours, it may return control to 503.

FIGS. 6(a) and 6(b) illustrate an embodiment of the feature extractionmodule configured in accordance with the present invention and is notintended to limit scope as one of ordinary skill would understand onreview of this application that other configurations could be utilizedwithout departing from the scope of the claimed invention. Referring toFIG. 6(a), the cognitive function module may access the recordings madeby the data collection module 400. Patterns in a user's interactionswith an electronic device are preferably identified and extracted fromthe data in 601-617. Each pattern has features and attributes that areextracted from the data. The features of each pattern include, forexample, timing, duration, frequency and amplitude, and number ofoccurrences. The attributes of each pattern include information relatingto the when, where, what, and why of each pattern. This data includes,for example, date-time of occurrence, GPS readings, accelerometer andgyroscope readings, physiology and kinetic readings, app in use at thetime.

In 601, recurring combinations of applications opened and closed by auser, frequency and latencies between opening and closing are preferablyidentified and extracted. In 603, patterns in a user's gestures, tappingand body motions such as head-tilting, swiping, or eye movements thatcan be used as commands to an electronic device are preferablyidentified and extracted from the recorded data. Patterns in falsepositive rates, scrolling during search and paging during browsing, arealso preferably extracted. Features of each pattern including duration,amplitude, frequency, and force used may be extracted when present inthe data or derived from the data. In 605, patterns in keys pressed on akeyboard or other input form-factor and in combination with gesture,tapping, body motion and eye movement inputs are preferably identifiedand extracted. For character inputs, recurring keystroke combinations,recurring spelling mistakes, omissions, back-space corrections,irregular latency variances in common words are preferably extracted.Features of each pattern including duration, frequency, latency, forceused, length of messages, and message coherence may be extracted fromthe data.

In 607, patterns in voice commands, words, and phrases used may beextracted. Following signal processing of the voice, recurringcombinations of phones and phoneme may be extracted. Features of eachpattern including voice pitch, amplitude, and frequency spectrum arepreferably extracted from the data. In 609, patterns inelectroencephalogram (EEG), locomotion, gait, and posture recorded fromwearable electronic devices are preferably identified and extracted.Features of each pattern derived from the EEG, accelerometer, gyroscope,and other sensor recordings are also preferably extracted.

In 611-617, features and patterns in behavioral activities recorded bymodule 400 are preferably extracted and used in the metric computationmodule 700 to explain temporal change in the values of the metrics ofcognitive function that are computed from input patterns and features601-609. In 611, patterns in words and phrases used to communicate inemail, text messaging, phone calls, and other communication modes may beextracted. Recurring sentiments in the user's communication and patternsin calls made/received, messages sent/received may also be extracted.Features of each pattern including frequency, duration, and recipientmay be extracted with the pattern. The patterns and features of 611 arepreferably a proxy to the user's level of social engagement.

In 613, a user's travel, patterns in the inferred mode of travelincluding vehicle, bicycle, foot or other, patterns in destinationpoints, features of those patterns derived from GPS data and 3^(rd)party data such as destination type like restaurant, shopping mall,museum, park, and features like speed of travel, time of day, day ofweek, date of year are preferably extracted. The patterns and featuresof 613 may be a proxy to the user's activity and in-person socialengagement.

Patterns in heart rate, blood pressure, body temperature, blood oxymetryand other physiologic measurements may be extracted from the data by615. Together with accelerometer and gyroscope data in 609, a rapidheart rate from anxiety or illness is preferably distinguished fromexercise-induced changes. Features of each pattern including maximum andminimum measurements, duration of pattern, frequency of pattern, arealso preferably extracted.

In 617, patterns in recorded daily caloric intake, nutritional intake byfood group or food type, intake of alcohol, caffeine, medicationsincluding prescription and non-prescription may be extracted. Featuressuch as time of day, total intake, and location of intake are preferablyderived or extracted from the recorded data.

The what, where, when, and why attributes extracted from the data foreach pattern in 601-617 may be used to filter those patterns prior toinputting into the metric computation module 700. They may also beincluded as covariates in the cognitive function model of 700. Forexample, including time of day or day of week may explain variance thatcan be attributed to individual fatigue and other factors that havenegative effects on cognitive function. GPS, gyroscope and accelerometerrecordings from module 400 may be used to correct for motion artifactfrom activities such as driving or walking. Lastly, the physiologicmeasurements of heart-rate, blood pressure, blood oxymetry, bodytemperature, and electroencephalogram recorded by module 400 may be usedto correct for the negative effects that anxiety, general malaise,illness, and fatigue have on cognitive function.

Referring to FIG. 6(b), statistics may be computed in 623 from thefeatures and attributes of the patterns 621 that were extracted in601-617. The statistics may be computed on the raw feature values orafter functional transformation of the feature values. The computedstatistics may also be computed on all available values of the featureor transformed feature, or they may be computed on a subset of valuesfiltered by one or more attributes. The statistics computed may include,for example, mean, median, mode, variance, kurtosis, moments, range,standard deviation, quantiles, inter-quantile ranges, and distributionparameters under different distributions such as exponential, normal, orlog-normal. Each computed statistic for each pattern, feature, andattribute combination is preferably outputted in 625.

Referring to FIG. 7(a), feature statistics 701 indexed by user, pattern,feature, statistic, and attribute for a population of users, and thecognitive function test scores 703 indexed by user and test for the sameuser population are preferably inputs to 705. The cognitive functiontests in 703 may include tests of general intelligence (IQ), memory,attention, executive function and processing speed. These tests mayinclude standardized neuropsychological tests such as the Wechsler AdultIntelligence Scale test, California Verbal Memory Test, Trail-Making(A&B) test, Wechsler Memory Scale-III test, Symbol Digit Modalitiestest, Wechsler Digit Span test, Conner's Continuous Performance test,Logical Memory test, Brief Visuospatial Memory test, Controlled OralWord Association (FAS Fluency) test, Animal Naming Test of VerbalFluency, and the Grooved Pegboard test. The tests may be administered bya psychometrician or taken on a computer. The scores on these tests arenumeric and are the inputs shown in 703. In 705, a cognitive functionmodel for each test in 703 is preferably learned using 701 as inputs tothe model and the score of the test 703 as the target output of themodel. One of several machine learning methods may be used to learn thetarget test scores 703 from the feature inputs 705. One suitable methodincludes a feedforward neural network with a single hidden layer ofneurons. To train the feedforward neural network, the input feature dataand output test score data is split into a training set, a validationset, and a testing set. The training set is used to learn the parametersof the neural network using the well-known backpropagation algorithmjointly with standard gradient descent to minimize the mean-squarederror between the target test scores and the predictions made by thefeedforward network on the input feature data. Training ends when themean-squared error of the validation data (not the training data)reaches a minimum to reduce the risk of over-fitting the data. The testdata is used to confirm that the trained network generalizes to newdata. Further testing can be achieved by cross-validation of the networkon the training data. Other machine learning methods may be suitable asany one of ordinary skill in the art will appreciate. The fittedcognitive function model may be outputted in 707 and generalizes to newinput feature data and outputs the test score of the neuropsychologytest for which it was trained. One of ordinary skill would understand onreview of this application that many functional models and fits of themodel are plausible to output a predictive model of user test scores 703from the user feature statistics 701.

Referring to FIG. 7(b), the output cognitive function models in 707 ofFIG. 7(a) are preferably used to output predictions of cognitivefunction test scores for the M tests on new input feature statistics 709indexed by user, pattern, feature, statistic, and attribute. The newinput feature statistics 709 may be for a new user for which cognitivefunction test scores are not available, or they may be featurestatistics of a user in the same population used to fit the model in 707but not the same feature statistics used in 707, for example, featurestatistics obtained at a later date under different circumstances forthat user. The cognitive function models for each test are preferablyevaluated in 711 on the new feature statistics and in 713 the scoreprediction for each cognitive function test is preferably outputtedtogether with α-confidence intervals around each prediction for abetween 0 and 1. The α-confidence intervals are calculated in a mannerknown and recognized in the art.

In addition to predicting scores of standard cognitive function tests,new cognitive function metrics may be computed from one or more metricfunctions in 717 of FIG. 7(c) using input feature statistics 715. Themetric function g has the property that it maps the input featurestatistics into a number. In 717, the cognitive function metric iscomputed from a user's feature statistics 715, a metric function g, andthe metric function value's percentile rank relative to a population ofusers. The cognitive function metric and percentile is outputted in 719.

Referring to FIG. 8, a user may create and access an online account toobtain aggregated and detailed views of recorded data, cognitivefunction evaluation and target recommendations to improve cognitivefunction. In 801, a user may create an account and input personaldemographic data and health data. The user also may set privacy levels,assign delegates to the account, or invite caregivers and physicians toparticipate. In 803, a user may access recent and historical views ofaggregated recordings of (i) mobility, (ii) activity, (iii) social, (iv)learning, (v) diet, (vi) medications, (vii) applications used, (viii)electroencephalogram (EEG), and (ix) physiology, for example. A calendarwidget (not illustrated) preferably enables the user to specify theaggregation period to use for the recent view and the historical views.Each group of recordings preferably offers granular views of behavior.Activity reports on physical activity and intensity, social engagementreports on outbound and inbound number of distinct people called,emailed, or text messaged, learning engagement reports on URLs visitedand ebooks read, subject type and language, diet reports on food andbeverages consumed and nutritional facts, applications reports on typeof application, application name, duration and frequency of use, EEGreports on electroencephalogram recordings, and physiology reports onheart rate, pulse oxymetry, and body temperature preferably by time ofday.

805 presents a time-series of the cognitive function evaluationscomputed by the metric computation module 700. If neuropsychologicalevaluations of cognitive function are available these may be overlaid onthe time-series in 805. Functional measures using blood-oxygen leveldependent (BOLD) functional magnetic resonance imaging (fMRI) andstructural volume estimates using MRI of brain regions responsible formotivation, memory, learning that include by way of example thecingulate cortex, hippocampus, and entorhinal cortex, may be furtheroverlaid on the time-series of 805.

The attribution report in 807 may compute the contribution of behavioralactivity inferred from the recordings by the data collection module 400to changes in cognitive function metrics. In 809, the attribution reportmay be used to set optimal target levels for mobility, physical, social,learning, diet, medications, and physiology to restore and improvecognitive function. To help the user achieve those targets, informationis preferably provided in 811 to direct the user to relevant groups,forums, and educational material. The cycle then repeats itself.

After a target interval of one or two weeks has passed, new behavioraltrend reports are preferably published in 803 and compared against theset targets, new values are computed for cognitive function and plottedin 805 to determine whether the target cognitive function was attained,new attributions are evaluated in 807 to explain observed changes incognitive function, and new targets are set in 809.

An individual on any given day of the week uses his or her mobile deviceor other electronic device regularly as a common way of communicating toothers, as a form of entertainment and as a means of retrieving andstoring information. Therefore, the technology as disclosed and claimedherein does not require supervision over the individual users and inorder to obtain meaningful information that is useful for measuringcognitive function and the individual does not have to change his/herlifestyle for the information to be gathered. For example, an individualmay select an application on a mobile computer to launch the applicationby tapping on an icon displayed as part of the user interface where theuser taps the touch-screen to open an application. The individual canlaunch an application such as a weather application to check the weatherforecast, launch an email application to check email, scroll throughemails and read and respond to some emails by keying in text with sometyping errors and some misspellings. The user may get up for an earlymorning walk or run or bike ride and a combination of information may becaptured by the global position function of the device as well asinformation or data from the gyroscopic motion sensor of the deviceindicative of the user exercising. When walking the user may trip orstumble quite often or their motion may not be fluid, which may becapture by the accelerometer and gyroscopic motion sensor as a suddenand violent movement. Also the user may actually trip and fall. The usermay go for a drive over what appears to be a familiar route, but the GPSfunction captures what appears to be a series of incorrect turns. Thismay be indicative of a problem, although it may also be captured thatthe individual is also placing a call, on a call, or sending a text atthe same time that the false or incorrect turn was made. The user mayopen a navigation application and enter a location search query and canuse touch-screen gestures to scroll around on map within an applicationand select items on the map for additional feature information. Whenscrolling through the map, the individual can touch and swipe the map tofollow highlighted directions or to search for a point of interest.Touches that dwell too long or swipes that are too short may result inan undesired response from the mobile computer. During the day, theindividual may be traveling to and fro alone, but may need to reach outto their spouse, or son or daughter to coordinate a pick-up or meeting.The user can type text messages to selectively reach out. The individualmay continue a text dialogue for a period of time. During the textdialogue the individual may have to back space several times to correcta typing error. The individual may want to meet with their spouse fordinner, but they do not know exactly were the restaurant of choice islocated. The user can use the voice command feature to requestdirections and the GPS can be tracking the user's location, the user canrequest other content by keying in on a keyboard presented by theapplication or use voice commands to do the same, read and scrollthrough content, and the user can make calls or listen to messages. Theuser's activity is recorded on the device's persistent storage. The usermakes or receives calls, email messages, and text messages. There canalso be several days in a row where the user is carrying his/herelectronic device, but out of the ordinary, the user may not place anycall or send any text messages, browse their favorite social mediaapplication or visit their favorite website.

The date and time of each of the above activities, the activityduration, and the sender and recipient phone number or email address maybe recorded in the device's persistent storage. The frequency of eachevent is preferably recorded, and the various qualitativecharacteristics of each event are also preferably recorded. The user maycarry the mobile device with them while driving, using publictransportation, walking, biking or running. While engaging in theseactivities, the electronic device's global positioning system (GPS) mayrecord the user's longitude and latitude coordinates. Similarly, duringthose activities the device's acceleration and gyroscopic motion along a3-coordinate system may be recorded. The type of locations that theindividual traveled may be determined and the characteristic of themotion of the user may also be evaluated for fluidity or erratic motion.This information is preferably recorded in the device's persistentstorage. When a user browses URLs on an internet browser applicationresident on the electronic device, or reads an e-book on an e-bookreader resident on the mobile device, the URLs browsed and the pages ofthe e-book read, the start time and end time between URLs and pages arepreferably recorded by the system and method of this invention andpersisted in the device's persistent storage.

Note that gesture, tapping, body motion and eye movement activity,typing activity, and voice command activity during the use ofapplications is preferably captured separately in and recorded with thetime and application in which that activity took place. In this way, thesystem tracks gestures used during browsing, paging and scrolling, forexample. Lastly, a bar code scanning application and medical reminderapplication resident on the electronic device may enable the user toscan grocery purchases, and meals and beverages purchased when thelatter have bar codes and to keep track of medications taken, dosage andfrequencies. The bar code scanning application preferably has access toa database of nutritional facts. If a nutritional fact for the scannedbar code is not in the database, then the application may instruct theuser to photograph the nutritional fact label of the item. The bar codeinformation and medications taken, dosages and frequencies arepreferably persisted in the device's persistent storage.

The data captured in the device's persistent storage may be transmittedto cloud computers. Transmission may use a secure channel and usehypertext transfer protocol secure (HTTPS) to securely transfer thedata. The cloud computers may analyze the recorded data againsthistorical recordings and against recordings averaged over other usersdemographically matched to the existing user. The output of the analysisis preferably an evaluation of cognitive function and the attribution tochanges in behavioral activities inferred from the activities recordedincluding social engagement, physical activity, learning activity, anddiet. The user may log into his or her personal password protectedonline account to view the results of the analysis together withsuggestions for improvement.

Various recordings of data representative of the quantity and quality ofinteractions and occurrences of events using the mobile computing devicemay be made. The data may be encoded with representative tags.

A description of the type of encoded data with representative logsemantics when an application is launched on the electronic device maybe as follows:

App Log Description app_pkg_name Application launched app_start_timeStart date and time of use app_end_time End date and time of use

A description of the type of encoded data with representative logsemantics following incoming, outgoing and missed calls on theelectronic device may be as follows:

Call Log Description call_type Values (1, 2, 3) meaning (incoming,outgoing, missed) phone_number Phone number of call call_start_timeStart date and time of call call_end_time End date and time of callaudio_recording_file File name of recorded call and null if not recorded

A description of the type of encoded data with representative logsemantics following gestures made on the touch-screen of the electronicdevice during use of an application may be as follows:

Gesture Log Description app_pkg_name Application in use view_name_in_pkgPackage within application (eg. Contacts or missed calls, in phoneapplication) scroll_unit Values (1, 2) meaning (items, pixels)item_count Number of scrollable items from_idx Index of first itemvisible when scrolling to_idx Index of last item visible when scrollingscroll_direction Values (1, 2) meaning scroll in (X, Y) directionmax_scroll Max_scroll x dir (or y dir): gives the max scroll offset ofthe source left edge (or top edge) in pixels scroll_value Scroll_value Xdir (or Y dir): gives the scroll offset of the source left edge (or topedge) in pixels timestamp Date and time

A description of the type of encoded GPS data with representative logsemantics on the electronic device may be as follows:

GPS Log Description latitude Latitude of current location longitudeLongitude of current location altitude Altitude of current locationbearing Horizontal direction of travel in degrees (0.0, 360.0] speedTravel speed in meters per second accuracy Radius in meters of 68%confidence circle timestamp Date and time

A description of the type of encoded data with representative logsemantics when the keyboard is used on the electronic device may be asfollows:

Key Log Description session_id Identifies each session key_type Values(1, 2) meaning (alphanumeric, control key) key_code Ascii code ofpressed key key_desc Either the alphanumeric value or a controldescriptor key_press_time Date and time in milliseconds when key ispressed key_release_time Date and time in milliseconds when key isreleased key_press_duration Duration of key press in milliseconds

A description of the type of encoded sensor data with representative logsemantics on a mobile electronic may be as follows:

Sensor Log Description sensor_type Values (1, 2) meaning (accelerometer,gyroscope) value_0 Value of first coordinate measurement value_1 Valueof second coordinate measurement value_2 Value of third coordinatemeasurement timestamp Date and time of coordinate measurements

A description of the type of encoded data with representative logsemantics when text messages are sent or received on the electronicdevice may be as follows:

SMS Log Description sms_type Values (1, 2) meaning (Incoming, Outgoing)phone_number Phone number of message message Message text timestamp Dateand time of message

A description of the type of encoded data with representative logsemantics when URLs are browsed on the electronic device may be asfollows:

URL Log Description URL URL viewed timestamp Start date and time of URLview

In one implementation of the method and system, in order to establish abaseline of data, supervised benchmark testing may be conducted on aninitial test group of individuals where these individuals take aneuropsychological benchmark test for cognitive function, and the datamay be stored. Each of the same individuals who are tested may beprovided with electronic devices having the computer program forperforming the system and method. Data for each individual may berecorded as outlined herein and the data from the electronic deviceusage may be correlated to the benchmark testing results and cognitivefunction. Cognitive function levels and bands may also be determinedfrom the result. Once certain baselines have been established andcorrelations are made between cognitive function and electronic deviceusage, subsequent electronic device usage by individuals may be utilizedto improve the system and method as learning occurs. The learning fromthe subsequent electronic device usage may be considered unsupervisedlearning.

The cognitive function module may access the recordings made by the datacollection module and transmitted to cloud computers. Patterns in auser's interactions with an electronic device captured are preferablyanalyzed for changes in cognitive function. The above described activityfor an individual and his interface with a electronic computing devicemay be captured and utilize. Changes in applications opened and closedby a user, frequency and latencies between opening and closing, andtheir diurnal and weekly variations may be inputs to the featureextraction module and the learning, and computation of cognitivefunction by the metric computation module. Changes in a user's gestures,tapping, body motions, and eye movements used as inputs to applicationssuch as type of gesture, gesture durations, including false positivegestures of excessive scrolling during search and excessive pagingduring browsing, may be additional inputs to the feature extractionmodule and the learning, and computation by the metric computationmodule. Changes in a user's kinetic activities of motion, gait andbalance recorded from wearable gyroscopic and accelerometer sensors, mayfurther be additional inputs to the feature extraction module and thelearning, and computation by the metric computation module.

Similarly, when the above described individual makes character inputs,recurring spelling mistakes, omissions, excessive backspace corrections,irregular latency variances in common words, length of messages, andmessage coherence may all be inputs into the feature extraction,learning, and computation of cognitive function. Signal processing ofspeech and voice may provide additional input to the computationincluding emerging irregularities in phones and phoneme latencies, andnarrowing or shifting of the voice frequency spectrum. The time of dayand day of week may be captured in the recordings of and used by thecognitive function computation to adjust, or explain variances that canbe attributed to individual fatigue and other factors that may haveshort-term effects on cognitive function.

Further, to correct for motion artifact such as from driving or walking,GPS, gyroscope and accelerometer recordings made are preferably used asadditional inputs in the feature extraction, learning and computation ofcognitive function. To correct for physiologic effects such as anxiety,general malaise, illness, the physiologic measurements of heart-rate,blood pressure, blood glucose, blood oxymetry, and body temperature whenavailable and recorded may be used as further inputs to the evaluationof cognitive function. Behavioral activities recorded on the electronicdevice may be analyzed to explain changes in the cognitive function thatis computed using inputs. A user's incoming and outgoing email, phonecalls, and text messages, their frequencies and length may also be usedas a proxy for the user's level of social engagement. A user's dailytravel, the inferred mode of travel including vehicle, bicycle, foot orother, the user's sleep and rest patterns inferred by electronic device“quiet” times, may be used to infer physical activity. When this data isavailable and when correlated with physical activity data such as rapidheart rates from anxiety or illness may be distinguished from exerciseinduced changes, improving the inference of physical activity andquantifying the intensity of that activity. Analysis of patterns ingroceries purchased, food and drinks consumed provides a proxy tonutritional intake may also be made. Analysis of changes in medicationstaken, dosages and frequencies including missed medications, changes indosage and new medications that may alter mental state are preferablyused to infer their effect on the user's cognitive function.

The various implementations and examples shown above illustrate a methodand system for assessing cognitive function using an electroniccomputing device. A user of the present method and system may choose anyof the above implementations, or an equivalent thereof, depending uponthe desired application. In this regard, it is recognized that variousforms of the subject method and system could be utilized withoutdeparting from the spirit and scope of the present implementation.

As is evident from the foregoing description, certain aspects of thepresent implementation are not limited by the particular details of theexamples illustrated herein, and it is therefore contemplated that othermodifications and applications, or equivalents thereof, will occur tothose skilled in the art. It is accordingly intended that the claimsshall cover all such modifications and applications that do not departfrom the spirit and scope of the present implementation. Accordingly,the specification and drawings are to be regarded in an illustrativerather than a restrictive sense.

Certain systems, apparatus, applications or processes are describedherein as including a number of modules. A module may be a unit ofdistinct functionality that may be presented in software, hardware, orcombinations thereof. When the functionality of a module is performed inany part through software, the module includes a computer-readablemedium. The modules may be regarded as being communicatively coupled.The inventive subject matter may be represented in a variety ofdifferent implementations of which there are many possible permutations.

The methods described herein do not have to be executed in the orderdescribed, or in any particular order. Moreover, various activitiesdescribed with respect to the methods identified herein can be executedin serial or parallel fashion. In the foregoing Detailed Description, itcan be seen that various features are grouped together in a singleembodiment for the purpose of streamlining the disclosure. This methodof disclosure is not to be interpreted as reflecting an intention thatthe claimed embodiments require more features than are expressly recitedin each claim. Rather, as the following claims reflect, inventivesubject matter may lie in less than all features of a single disclosedembodiment. Thus, the following claims are hereby incorporated into theDetailed Description, with each claim standing on its own as a separateembodiment.

In an example embodiment, the machine operates as a standalone device ormay be connected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of a server or aclient machine in server-client network environment, or as a peermachine in a peer-to-peer (or distributed) network environment. Themachine may be a server computer, a client computer, a personal computer(PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant(PDA), a cellular telephone, a smart phone, a web appliance, a networkrouter, switch or bridge, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine or computing device. Further, while only a singlemachine is illustrated, the term “machine” shall also be taken toinclude any collection of machines that individually or jointly executea set (or multiple sets) of instructions to perform any one or more ofthe methodologies discussed herein.

The example computer system and client computers include a processor(e.g., a central processing unit (CPU) a graphics processing unit (GPU)or both), a main memory and a static memory, which communicate with eachother via a bus. The computer system may further include avideo/graphical display unit (e.g., a liquid crystal display (LCD) or acathode ray tube (CRT)). The computer system and client computingdevices also include an alphanumeric input device (e.g., a keyboard ortouch-screen), a cursor control device (e.g., a mouse or gestures on atouch-screen), a drive unit, a signal generation device (e.g., a speakerand microphone) and a network interface device.

The drive unit includes a computer-readable medium on which is storedone or more sets of instructions (e.g., software) embodying any one ormore of the methodologies or systems described herein. The software mayalso reside, completely or at least partially, within the main memoryand/or within the processor during execution thereof by the computersystem, the main memory and the processor also constitutingcomputer-readable media. The software may further be transmitted orreceived over a network via the network interface device.

The term “computer-readable medium” should be taken to include a singlemedium or multiple media (e.g., a centralized or distributed database,and/or associated caches and servers) that store the one or more sets ofinstructions. The term “computer-readable medium” shall also be taken toinclude any medium that is capable of storing or encoding a set ofinstructions for execution by the machine and that cause the machine toperform any one or more of the methodologies of the presentimplementation. The term “computer-readable medium” shall accordingly betaken to include, but not be limited to, solid-state memories, andoptical media, and magnetic media.

I claim:
 1. A computer-implemented method for assessing brain health ofa person, the method comprising the steps of: a. recording, separatelyand unobtrusively at a data collection module installed at one or moreelectronic computing devices, inputs associated with one or moreinteractions of the person with said one or more electronic computingdevices; b. transmitting said inputs from said data collection moduleinstalled at the one or more electronic computing devices to a computingsystem comprising a processor executing a learning function mapping; c.computing, by said computing system, a brain health metric by performingsaid learning function mapping from said inputs to said brain healthmetric, wherein said brain health metric is a neuropsychological test;and d. outputting, by said computing system, said brain health metric.2. The method of claim 1, wherein the one or more interactions includesat least one of: applications opened, inputs typed, gesture patternsused on a touch screen, body motions, eye movements, and voice input. 3.The method of claim 1, wherein the one or more electronic devicesincludes at least one of: a smart phone, a tablet computer, a wearableelectronic device, and a household electronic device.
 4. The method ofclaim 2, where to quantify the effect of social interaction, mobility,physiology, cognitive stimulation, diet, medication taken and traumaticexposure on a person's brain health, said method further comprises thesteps of: a. recording at said one or more electronic computing devicesinputs associated with a measurement of one of social interaction,mobility, physiology, cognitive stimulation, diet, medication taken andtraumatic exposure; b. performing a learning function mapping from saidinputs associated with the measurement to said brain health metric; andc. outputting an attribution to said brain health metric that isexplained by said inputs associated with the measurement.
 5. Thecomputer implemented method of claim 4, wherein the social interactionmeasurement is one of length, duration, frequency, volume, sentiment,and mood associated with one of an incoming call, an outgoing call, atext, and an email message.
 6. The computer implemented method of claim4, wherein the social interaction measurement is a measurement from oneof an email, phone, texting, or other communication application on saidone or more electronic computing devices.
 7. The computer implementedmethod of claim 4, wherein the mobility measurement is one of intensity,duration, and frequency of locomotor activity.
 8. The computerimplemented method of claim 4, wherein the mobility measurement is ameasurement from one of an accelerometer application and a gyroscopeapplication on said one or more electronic computing devices.
 9. Thecomputer implemented method of claim 4, wherein the physiologymeasurement is one of a heart rate, a blood pressure, a blood oxymetry,a blood glucose, a body temperature, a body fat, a body weight, a sleepduration and quality, and an electroencephalogram.
 10. The computerimplemented method of claim 4, wherein the cognitive stimulationmeasurement is one of URLs visited, books and articles read, and gamesplayed on said one or more electronic computing devices.
 11. Thecomputer implemented method of claim 4, wherein the cognitivestimulation measurement is one of a setting of, or a recording from, abrain stimulation device.
 12. The computer implemented method of claim4, wherein the diet measurement is one of a caloric intake, anutritional value, a food type, and alcohol, caffeine, drug, and vitaminconsumption.
 13. The computer implemented method of claim 4, wherein themedication measurement is one of a dosage, a frequency, and a durationof a medication.
 14. The computer implemented method of claim 4, whereinthe traumatic exposure measurement is one of a date and a severity of anincident.
 15. A computer-readable medium comprising instructions thatwhen executed by a processor perform a method for assessing brain healthof a person, the method comprising the following steps: a. recording,separately and unobtrusively at a data collection module installed atone or more electronic computing devices, inputs associated with one ormore interactions of the person with said one or more electroniccomputing devices; b. transmitting said inputs from said data collectionmodule installed at the one or more electronic computing devices to acomputing system executing a learning function mapping; c. computing, bysaid computing system, a brain health metric by performing said learningfunction mapping from said inputs to said brain health metric, whereinsaid brain health metric is a neuropsychological test; and d.outputting, by said computing system, said brain health metric.
 16. Thecomputer-readable medium of claim 15, wherein the one or moreinteractions includes at least one of: applications opened, inputstyped, gesture patterns used on a touch screen, body motions, eyemovements, and voice input.
 17. The computer-readable medium of claim15, wherein the one or more electronic devices includes at least one of:a smart phone, a tablet computer, a wearable electronic device, and ahousehold electronic device.
 18. The computer-readable medium of claim17, where to quantify an effect of social interaction, mobility,physiology, cognitive stimulation, diet, medication taken and traumaticexposure on a person's brain health, said instructions that whenexecuted by a processor further comprise the steps of: a. recording atsaid one or more electronic computing devices inputs associated with ameasurement of one of social interaction, mobility, physiology,cognitive stimulation, diet, medication taken and traumatic exposure; b.performing a learning function mapping from said inputs associated withthe measurement to said brain health metric; and c. outputting anattribution to said brain health metric that is explained by said inputsassociated with the measurement.
 19. A computer-implemented system forassessing brain health of a person, the system comprising: a. one ormore electronic computing devices: recording, separately andunobtrusively at a data collection module, inputs associated with one ormore interactions of the person with said one or more electroniccomputing devices; and transmitting said inputs from said datacollection module installed at the one or more electronic computingdevices to a computing system comprising a processor executing alearning function mapping; b. said computing system: computing a brainhealth metric by performing said learning function mapping from saidinputs to said brain health metric, wherein said brain health metric isa neuropsychological test; and outputting said brain health metric.